<!-- build time:Wed Jun 21 2023 22:33:33 GMT+0800 (GMT+08:00) --><!DOCTYPE html><html lang="zh-CN"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=2"><meta name="theme-color" content="#FFF"><meta name="baidu-site-verification" content="code-C0oocRvMWv"><link rel="apple-touch-icon" sizes="180x180" href="/images/apple-touch-icon.png"><link rel="icon" type="image/ico" sizes="32x32" href="/images/favicon.ico"><link rel="mask-icon" href="/images/logo.svg" color=""><link rel="manifest" href="/images/manifest.json"><meta name="msapplication-config" content="/images/browserconfig.xml"><meta http-equiv="Cache-Control" content="no-transform"><meta http-equiv="Cache-Control" content="no-siteapp"><meta name="baidu-site-verification" content="https://jiang-hs.gitee.io"><link rel="alternate" type="application/rss+xml" title="航 順" href="https://jiang-hs.gitee.io/rss.xml"><link rel="alternate" type="application/atom+xml" title="航 順" href="https://jiang-hs.gitee.io/atom.xml"><link rel="alternate" type="application/json" title="航 順" href="https://jiang-hs.gitee.io/feed.json"><link rel="stylesheet" href="//fonts.googleapis.com/css?family=Mulish:300,300italic,400,400italic,700,700italic%7CFredericka%20the%20Great:300,300italic,400,400italic,700,700italic%7CNoto%20Serif%20JP:300,300italic,400,400italic,700,700italic%7CNoto%20Serif%20SC:300,300italic,400,400italic,700,700italic%7CInconsolata:300,300italic,400,400italic,700,700italic&display=swap&subset=latin,latin-ext"><link rel="stylesheet" href="/css/app.css?v=0.0.0"><meta name="keywords" content="推荐系统,新闻推荐,论文精读,预训练模型"><link rel="canonical" href="https://jiang-hs.gitee.io/posts/c51ea00e/"><meta name="description" content="新闻推荐需要对新闻文章的底层语义进行深入的洞察。因此，预先训练的语言模型 (PLMs)，如 BERT 和 RoBERTa，可以极大地提高推荐质量。然而，将新闻推荐器与如此大的模型一起训练是非常具有挑战性的：新闻推荐器的学习需要密集的新闻编码操作，如果使用 PLMs 作为新闻编码器，其成本是不可承受的。在本文中，我们提出了一个新的框架，SpeedyFeed，它可以高效地训练基于 plms 的高质量"><meta property="og:type" content="article"><meta property="og:title" content="SpeedyFeed：用预训练语言模型训练大规模新闻推荐器"><meta property="og:url" content="https://jiang-hs.gitee.io/posts/c51ea00e/index.html"><meta property="og:site_name" content="航 順"><meta property="og:description" content="新闻推荐需要对新闻文章的底层语义进行深入的洞察。因此，预先训练的语言模型 (PLMs)，如 BERT 和 RoBERTa，可以极大地提高推荐质量。然而，将新闻推荐器与如此大的模型一起训练是非常具有挑战性的：新闻推荐器的学习需要密集的新闻编码操作，如果使用 PLMs 作为新闻编码器，其成本是不可承受的。在本文中，我们提出了一个新的框架，SpeedyFeed，它可以高效地训练基于 plms 的高质量"><meta property="og:locale" content="zh_CN"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230420151207.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408203243.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408203648.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408205058.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408205534.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408213556.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408214518.png"><meta property="og:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408215046.png"><meta property="article:published_time" content="2023-04-15T01:40:08.000Z"><meta property="article:modified_time" content="2023-04-21T06:00:36.099Z"><meta property="article:author" content="hang shun"><meta property="article:tag" content="推荐系统"><meta property="article:tag" content="新闻推荐"><meta property="article:tag" content="论文精读"><meta property="article:tag" content="预训练模型"><meta name="twitter:card" content="summary"><meta name="twitter:image" content="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230420151207.png"><title>SpeedyFeed：用预训练语言模型训练大规模新闻推荐器 - 用预训练模型做新闻推荐 - 新闻推荐 - 论文精读 | hang shun = 航 順 = 天官赐福，百无禁忌</title><meta name="generator" content="Hexo 5.4.2"></head><body itemscope itemtype="http://schema.org/WebPage"><div id="loading"><div class="cat"><div class="body"></div><div class="head"><div class="face"></div></div><div class="foot"><div class="tummy-end"></div><div class="bottom"></div><div class="legs left"></div><div class="legs right"></div></div><div class="paw"><div class="hands left"></div><div class="hands right"></div></div></div></div><div id="container"><header id="header" itemscope itemtype="http://schema.org/WPHeader"><div class="inner"><div id="brand"><div class="pjax"><h1 itemprop="name headline">SpeedyFeed：用预训练语言模型训练大规模新闻推荐器</h1><div class="meta"><span class="item" title="创建时间：2023-04-15 09:40:08"><span class="icon"><i class="ic i-calendar"></i> </span><span class="text">发表于</span> <time itemprop="dateCreated datePublished" datetime="2023-04-15T09:40:08+08:00">2023-04-15</time> </span><span class="item" title="本文字数"><span class="icon"><i class="ic i-pen"></i> </span><span class="text">本文字数</span> <span>5.6k</span> <span class="text">字</span> </span><span class="item" title="阅读时长"><span class="icon"><i class="ic i-clock"></i> </span><span class="text">阅读时长</span> <span>5 分钟</span></span></div></div></div><nav id="nav"><div class="inner"><div class="toggle"><div class="lines" aria-label="切换导航栏"><span class="line"></span> <span class="line"></span> <span class="line"></span></div></div><ul class="menu"><li class="item title"><a href="/" rel="start">hang shun</a></li></ul><ul class="right"><li class="item theme"><i class="ic i-sun"></i></li><li class="item search"><i class="ic i-search"></i></li></ul></div></nav></div><div id="imgs" class="pjax"><ul><li class="item" data-background-image="https://pic1.imgdb.cn/item/60d7fbf45132923bf8b6f1f1.jpg"></li><li class="item" data-background-image="https://pic1.imgdb.cn/item/60d7f93d5132923bf8a86d6c.jpg"></li><li class="item" data-background-image="https://pic1.imgdb.cn/item/60d7f95f5132923bf8a924a8.jpg"></li><li class="item" data-background-image="https://pic1.imgdb.cn/item/60d7fbfa5132923bf8b712a9.jpg"></li><li class="item" data-background-image="https://pic1.imgdb.cn/item/64427dbc0d2dde5777b205ff.webp"></li><li class="item" data-background-image="https://pic1.imgdb.cn/item/60d7f9cb5132923bf8ab5c63.jpg"></li></ul></div></header><div id="waves"><svg class="waves" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 24 150 28" preserveAspectRatio="none" shape-rendering="auto"><defs><path id="gentle-wave" d="M-160 44c30 0 58-18 88-18s 58 18 88 18 58-18 88-18 58 18 88 18 v44h-352z"/></defs><g class="parallax"><use xlink:href="#gentle-wave" x="48" y="0"/><use xlink:href="#gentle-wave" x="48" y="3"/><use xlink:href="#gentle-wave" x="48" y="5"/><use xlink:href="#gentle-wave" x="48" y="7"/></g></svg></div><main><div class="inner"><div id="main" class="pjax"><div class="article wrap"><div class="breadcrumb" itemscope itemtype="https://schema.org/BreadcrumbList"><i class="ic i-home"></i> <span><a href="/">首页</a></span><i class="ic i-angle-right"></i> <span itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" itemprop="item" rel="index" title="分类于 论文精读"><span itemprop="name">论文精读</span></a><meta itemprop="position" content="1"></span><i class="ic i-angle-right"></i> <span itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" itemprop="item" rel="index" title="分类于 新闻推荐"><span itemprop="name">新闻推荐</span></a><meta itemprop="position" content="2"></span><i class="ic i-angle-right"></i> <span class="current" itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/%E7%94%A8%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B%E5%81%9A%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" itemprop="item" rel="index" title="分类于 用预训练模型做新闻推荐"><span itemprop="name">用预训练模型做新闻推荐</span></a><meta itemprop="position" content="3"></span></div><article itemscope itemtype="http://schema.org/Article" class="post block" lang="zh-CN"><link itemprop="mainEntityOfPage" href="https://jiang-hs.gitee.io/posts/c51ea00e/"><span hidden itemprop="author" itemscope itemtype="http://schema.org/Person"><meta itemprop="image" content="/images/avatar.jpg"><meta itemprop="name" content="hang shun"><meta itemprop="description" content="天官赐福，百无禁忌, 世中逢尔，雨中逢花"></span><span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization"><meta itemprop="name" content="航 順"></span><div class="body md" itemprop="articleBody"><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230420151207.png" alt="" width="80%"></p><p>新闻推荐需要对新闻文章的底层语义进行深入的洞察。因此，预先训练的语言模型 (PLMs)，如 BERT 和 RoBERTa，可以极大地提高推荐质量。然而，将新闻推荐器与如此大的模型一起训练是非常具有挑战性的：新闻推荐器的学习需要密集的新闻编码操作，如果使用 PLMs 作为新闻编码器，其成本是不可承受的。在本文中，我们提出了一个新的框架，SpeedyFeed，它可以高效地训练基于 plms 的高质量新闻推荐器。SpeedyFeed 的重点在于其轻量级编码管道，这带来了三个主要优势。首先，它使训练工作流的中间结果完全可重用，从而消除了大部分重复但冗余的编码操作。其次，它提高了训练工作流的数据效率，可以从编码中消除非信息性数据。第三，利用简化的新闻编码和紧凑的新闻表示，进一步节省了成本。大量实验表明，SpeedyFeed 可使训练过程加速 100 倍以上，使大模型在海量用户数据上得到高效有效的训练。来自 SpeedyFeed 的训练有素的基于 plms 的模型展示了极具竞争力的性能，在这方面它优于最先进的新闻推荐，利润率很高。SpeedyFeed 也是一个模型不可知的框架，它可能适用于广泛的基于内容的推荐系统；因此，整个框架是开源的，以促进相关领域的进展。</p><h1 id="介绍"><a class="anchor" href="#介绍">#</a> 介绍</h1><p>网络新闻平台已经成为重要的信息获取媒体。面对海量的网络新闻，个性化的新闻推送势在必行，用户可以通过个性化的新闻推送获取自己感兴趣的新闻。高质量的新闻推荐建立在对新闻文章底层语义的准确理解之上。因此，BERT 和 RoBERTa [4]、[5] 等预训练语言模型 (PLMs) 在一般文本理解任务中表现出色，是应用于新闻编码器的理想选择。然而，plm 对新闻推荐人的端到端培训并不是很友好。<strong>一方面，使用 PLMs 是昂贵的：编码速度相对较慢，考虑到 PLMs 的相当大的尺寸，GPU RAM 消耗将是巨大的。另一方面，新闻推荐器的训练需要密集的新闻编码操作：为了从用户的每个点击信号中学习，需要对用户的整个历史新闻点击进行编码，如果使用 PLMs，其计算成本将是令人望而却步的。因此，基于 plms 的新闻推荐器的发展受到了效率瓶颈的严重限制。</strong></p><h1 id="相关工作"><a class="anchor" href="#相关工作">#</a> 相关工作</h1><h2 id="预训练模型"><a class="anchor" href="#预训练模型">#</a> 预训练模型</h2><p>提出了预先训练的语言模型，利用在大规模语料库上训练的神经网络学习通用表示 / 生成模型。早期的工作是从一些浅层结构开始的，如 Skip-Gram [17] 和 GloVe [18]; 近年来，网络结构正在迅速扩大：从 EMLo [19]， GPT [20]，到 BERT [5]， RoBERTa [5]， UniLM [21]，直到今天的 GPT-3 [22]。大规模模型经过大量语料库的充分训练，在一般 NLP 任务上表现出优越的能力，例如语义匹配、问题回答、机器翻译和响应生成。</p><p>预训练的语言模型也被大量应用于检索或信息过滤相关场景 [23]，[24]，[25]; 例如，在 [26] 中，PLMs 被训练为知识检索，而在 [27] 中，PLMs 被微调为广告关键字匹配。在这些场景中，plm 需要将查询和关键字表示到它们的潜在嵌入中，其中查询 - 关键字关系可以通过它们的嵌入相似性反映出来。显然，新闻推荐也是类似的应用。然而，基于 PLMs 的新闻推荐可能相对更昂贵：为了将用户与候选新闻匹配，它需要使用 PLMs 对用户的所有历史新闻点击进行编码，这将导致巨大的编码成本。</p><h1 id="方法"><a class="anchor" href="#方法">#</a> 方法</h1><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408203243.png" alt="" width="60%"></p><p>算法 1 给出了轻量级编码管道的主要逻辑。对于每个小批，我们应用集中式新闻编码 (第 4.1.1 节)，从用户张量和新闻张量中收集所有涉及的新闻文章到合并集 m 中。然后从缓存的新闻嵌入 MC 中以查找速率 pt 采样查找集 ML，查找集中的新闻文章直接使用其缓存的嵌入，记为 Θ1。查找集之外的新闻文章是用 BusLM 编码的 (第 4.1.3 节)，其中提供 Θ2。整个新闻嵌入 Θ1∪Θ2 被分派到它们的原始位置 (在用户历史或候选新闻中); 然后，使用新生成的新闻嵌入 Θ2 刷新缓存。最后，使用自回归用户建模 (第 4.1.4 节) 计算总体预测损失，如式 5 所示。</p><h2 id="集中式新闻编码"><a class="anchor" href="#集中式新闻编码">#</a> 集中式新闻编码</h2><p>在典型的训练工作流中，新闻编码器将直接对输入张量 (即用户张量，新闻张量) 进行新闻嵌入。在此过程中，填充新闻与有效新闻一起编码，导致数据效率较低。</p><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408203648.png" alt="" width="80%"></p><p><strong>与典型的方法不同，一个公共小批中的所有新闻文章都联合编码在 SpeedyFeed 中 (如图 2 所示)</strong>。集中式新闻编码需要 3 个步骤：收集、编码和分发。一旦给出一个小批量，它就会将来自所有用户和候选人的新闻文章收集到一个合并集中。填充的新闻和复制的新闻都被删除。然后，为合并集中的所有剩余新闻生成新的嵌入。最后，将新闻嵌入发送到它们原来的训练实例。注意，填充的新闻文章也需要它们的嵌入，以便推断用户嵌入；在这个地方，一个虚拟向量被插入填充新闻所占据的位置，因此不需要额外的编码成本。</p><h2 id="缓存加速新闻编码"><a class="anchor" href="#缓存加速新闻编码">#</a> 缓存加速新闻编码</h2><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408205058.png" alt="" width="80%"></p><p>为了充分复用媒体间新闻编码结果，开发了缓存机制。特别是，微软新闻的一个值得注意的观察是它的新闻点击分布的长尾属性。如表 1 所示，前<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">1\%</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.80556em;vertical-align:-.05556em"></span><span class="mord">1</span><span class="mord">%</span></span></span></span> 的热门新闻文章产生了近<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>60</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">60\%</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.80556em;vertical-align:-.05556em"></span><span class="mord">6</span><span class="mord">0</span><span class="mord">%</span></span></span></span> 的新闻点击量。因此，这种受欢迎的新闻文章可能广泛存在于大多数用户的历史记录中，使得它们经常在不同的训练批次中重新编码。知道模型参数更新的学习率相当小，通常在<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi>e</mi><mtext>−</mtext><mn>5</mn></mrow><annotation encoding="application/x-tex">1e−5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.72777em;vertical-align:-.08333em"></span><span class="mord">1</span><span class="mord mathnormal">e</span><span class="mord">−</span><span class="mord">5</span></span></span></span> 量级，一篇新闻文章最近的嵌入可以在当前的迷你批处理中重用，以进行近似。基于这种直觉，我们提出了缓存加速新闻编码，在内存中维护一个缓存，用于存储新的新闻嵌入。新闻编码工作流相应地改变，如图 3 所示。</p><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408205534.png" alt="" width="80%"></p><p>图 3.cache 加速新闻编码说明：通过查找缓存获得缓存内新闻文章的嵌入；非缓存的新闻文章的嵌入需要从头编码。</p><ul><li><strong>News Encoding with Cache.</strong> 对于一个小批量中的每一篇新闻，训练器会首先检查缓存：如果缓存中嵌入了新闻的副本，它将直接重用而不进行编码；否则，新闻文章将从头编码。</li></ul><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408213556.png" alt="" width="80%"></p><p>图 4. 缓存管理说明：在初始阶段，以概率<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">p_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span></span></span></span> 查找缓存的新闻嵌入；缓存的新闻嵌入将在 γ 步后过期；所有设备将共享一个维护在主节点内存中的公共缓存。</p><ul><li><strong>Cache Management Policy.</strong> 缓存管理遵循以下原则。首先，缓存中的所有嵌入都必须在过去的几个步骤中新生成；否则，它将与当前模型不兼容。其次，动态调度缓存查找：在初始阶段，使用缓存的新闻嵌入的概率相对较低，因为模型参数的逐级变化比较剧烈；随着训练的进行，查找率应该逐渐提高，因为模型参数的变化变得比较温和。</li></ul><p>基于上述原则，制定了缓存管理策略 (如图 4 所示)，该策略受两个决定性变量的影响：逐步查找速率<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">p_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span></span></span></span> 和过期步骤<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi></mrow><annotation encoding="application/x-tex">γ</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05556em">γ</span></span></span></span>。1) 使用一个指数调度器来控制是否查找缓存的概率：当训练开始时，以 0 的概率查找缓存；在第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.61508em;vertical-align:0"></span><span class="mord mathnormal">t</span></span></span></span> 步，查找概率会逐渐增长到<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">p_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span></span></span></span>。</p><p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable width="100%"><mtr><mtd width="50%"></mtd><mtd><mrow><msub><mi>p</mi><mi>t</mi></msub><mo>=</mo><mn>1.0</mn><mo>−</mo><mi>exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo>−</mo><mi>β</mi><mi>t</mi><mo stretchy="false">)</mo></mrow></mtd><mtd width="50%"></mtd><mtd><mtext>(2)</mtext></mtd></mtr></mtable><annotation encoding="application/x-tex">p_t=1.0-\exp(-\beta t) \tag2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:.2777777777777778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:.2777777777777778em"></span></span><span class="base"><span class="strut" style="height:.72777em;vertical-align:-.08333em"></span><span class="mord">1</span><span class="mord">.</span><span class="mord">0</span><span class="mspace" style="margin-right:.2222222222222222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:.2222222222222222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mop">exp</span><span class="mopen">(</span><span class="mord">−</span><span class="mord mathnormal" style="margin-right:.05278em">β</span><span class="mord mathnormal">t</span><span class="mclose">)</span></span><span class="tag"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mord text"><span class="mord">(</span><span class="mord"><span class="mord">2</span></span><span class="mord">)</span></span></span></span></span></span></p><p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>β</mi></mrow><annotation encoding="application/x-tex">β</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.8888799999999999em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05278em">β</span></span></span></span> 是生长速率的超参数，它让<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">p_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span></span></span></span> 在训练过程的初始阶段后增长到 1.0。2) 一个缓存的新闻嵌入在<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi></mrow><annotation encoding="application/x-tex">γ</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.625em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05556em">γ</span></span></span></span> 步骤后过期，然后从缓存中删除。</p><p>最后，我们不再为每个训练线程维护一个私有缓存，而是在主节点中建立一个全局缓存。这样，在一个节点上新生成的新闻嵌入可以在所有设备上共享，从而适应新闻推荐人员的分布式训练。此外，缓存维护在内存中，而不是 GPU RAM；因此，它几乎是免费的，而且存储容量可以大到足以容纳所有所需的嵌入。</p><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408214518.png" alt="" width="80%"></p><p>我们在算法 2 中总结了缓存机制。在训练步骤 t 中，我们通过集中新闻编码输入一个合并的新闻集 M。首先，根据当前步骤 t 和超参数 β(第 3 行) 生成查找概率 pt。我们使用一个随机值来决定是否从缓存中读取嵌入。如果为真，对于缓存中的所有新闻嵌入，我们只加载在当前步骤 (第 8 行) 之前小于 γ 训练步骤编码的新闻嵌入。这些加载的嵌入用 Θ1 表示，对应的新闻用 ML 表示。对于没有缓存的新闻，即 M \ ML，我们用 BusLM 将其编码为 Θ2 嵌入，这将在下一个子集中介绍，并将 Θ2 写入缓存。最后，Θ1∪Θ2 是步骤 t 中全新的嵌入。</p><h2 id="总线语言模型"><a class="anchor" href="#总线语言模型">#</a> 总线语言模型</h2><p>我们进一步分析了如何以经济的方式进行新闻编码。新闻编码复杂度是新闻长度 O (n2) 的平方。一种简单的时间缩减方法是将新闻分成几个子组件，例如标题、摘要和正文，如 [15]; 文本段被独立处理，其编码结果将被聚合，用于最终的新闻嵌入。如果可以将文本分割成<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.68333em;vertical-align:0"></span><span class="mord mathnormal" style="margin-right:.07153em">K</span></span></span></span> 个 “几乎等长” 的段，则该操作可以将时间复杂度降低到<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>N</mi><mn>2</mn></msup><mi mathvariant="normal">/</mi><mi>K</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N^{2}/K)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.064108em;vertical-align:-.25em"></span><span class="mord mathnormal" style="margin-right:.02778em">O</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:.10903em">N</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.8141079999999999em"><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span><span class="mord">/</span><span class="mord mathnormal" style="margin-right:.07153em">K</span><span class="mclose">)</span></span></span></span>。然而，单纯的文本分割会影响新闻嵌入质量，因为文本片段在编码过程中无法相互引用。</p><p>受高效 transformers 最新进展的启发，我们提出了<strong> BusLM</strong> (图 5) 来编码新闻文章，其中实现了完全保留嵌入质量的加速。在<strong> BusLM</strong> 中，将输入的新闻统一划分为<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.68333em;vertical-align:0"></span><span class="mord mathnormal" style="margin-right:.07153em">K</span></span></span></span> 个文本段，使编码复杂度降低到<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>N</mi><mn>2</mn></msup><mi mathvariant="normal">/</mi><mi>K</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N^{2}/K)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.064108em;vertical-align:-.25em"></span><span class="mord mathnormal" style="margin-right:.02778em">O</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:.10903em">N</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.8141079999999999em"><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span><span class="mord">/</span><span class="mord mathnormal" style="margin-right:.07153em">K</span><span class="mclose">)</span></span></span></span>。片段仍然由 transformers 编码；然而，在 transformers 之间建立了分层的 “总线连接”，这使得信息可以跨段交换。</p><p><img data-src="https://shun309.oss-cn-hangzhou.aliyuncs.com/photos/20230408215046.png" alt="" width="80%"></p><p>图 5.<strong>BusLm</strong> 的图示 (使用第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 层进行演示)：来自所有段的第一个令牌被收集为第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 层的总线<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mi>u</mi><msup><mi>s</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">Bus^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.824664em;vertical-align:0"></span><span class="mord mathnormal" style="margin-right:.05017em">B</span><span class="mord mathnormal">u</span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.824664em"><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>；<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mi>u</mi><msup><mi>s</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">Bus^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.824664em;vertical-align:0"></span><span class="mord mathnormal" style="margin-right:.05017em">B</span><span class="mord mathnormal">u</span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.824664em"><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> 被广播到所有段，使得对于每个段，transformers 兼顾段内元素和总线元素。</p><p>在转换器的每一层中，为每个段选择一个 “代理嵌入”，作为其底层信息的草图。为了尽可能避免额外的计算，我们直接选择每个片段的第一个嵌入作为它的代理；例如，对于第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 层的片段<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.85396em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05724em">j</span></span></span></span>，选择<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi mathvariant="bold">H</mi><mi>j</mi><mi>i</mi></msubsup><mo stretchy="false">[</mo><mn>0</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathbf{H}_{j}^{i}[0]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.219436em;vertical-align:-.394772em"></span><span class="mord"><span class="mord"><span class="mord mathbf">H</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.824664em"><span style="top:-2.441336em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span></span></span></span><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.394772em"><span></span></span></span></span></span></span><span class="mopen">[</span><span class="mord">0</span><span class="mclose">]</span></span></span></span> 作为代理 (设<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi mathvariant="bold">H</mi><mi>j</mi><mi>i</mi></msubsup></mrow><annotation encoding="application/x-tex">\mathbf{H}_{j}^{i}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.219436em;vertical-align:-.394772em"></span><span class="mord"><span class="mord"><span class="mord mathbf">H</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.824664em"><span style="top:-2.441336em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span></span></span></span><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.394772em"><span></span></span></span></span></span></span></span></span></span> 是第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.85396em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05724em">j</span></span></span></span> 个片段在第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 层上的嵌入序列)。来自所有网段的第 i 层的代理嵌入被收集为第<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 条总线：</p><p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable width="100%"><mtr><mtd width="50%"></mtd><mtd><mrow><msup><mrow><mi mathvariant="bold">B</mi><mi mathvariant="bold">u</mi><mi mathvariant="bold">s</mi></mrow><mi>i</mi></msup><mo>=</mo><mo stretchy="false">{</mo><msubsup><mi mathvariant="bold">H</mi><mi>j</mi><mi>i</mi></msubsup><mo stretchy="false">[</mo><mn>0</mn><mo stretchy="false">]</mo><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></msubsup></mrow></mtd><mtd width="50%"></mtd><mtd><mtext>(3)</mtext></mtd></mtr></mtable><annotation encoding="application/x-tex">\mathbf{Bus}^i=\{\mathbf{H}^i_j[0]\}_{j=1}^K\tag3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.900674em;vertical-align:0"></span><span class="mord"><span class="mord"><span class="mord mathbf">B</span><span class="mord mathbf">u</span><span class="mord mathbf">s</span></span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.900674em"><span style="top:-3.1390100000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:.2777777777777778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:.2777777777777778em"></span></span><span class="base"><span class="strut" style="height:1.274439em;vertical-align:-.383108em"></span><span class="mopen">{</span><span class="mord"><span class="mord"><span class="mord mathbf">H</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.874664em"><span style="top:-2.4530000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span></span></span><span style="top:-3.1130000000000004em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.383108em"><span></span></span></span></span></span></span><span class="mopen">[</span><span class="mord">0</span><span class="mclose">]</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.891331em"><span style="top:-2.4530000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.1130000000000004em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:.07153em">K</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.383108em"><span></span></span></span></span></span></span></span><span class="tag"><span class="strut" style="height:1.283782em;vertical-align:-.383108em"></span><span class="mord text"><span class="mord">(</span><span class="mord"><span class="mord">3</span></span><span class="mord">)</span></span></span></span></span></span></p><p>总线被用作信息交换的媒介，所有的分段都可以直接参与。具体地说，第 <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.85396em;vertical-align:-.19444em"></span><span class="mord mathnormal" style="margin-right:.05724em">j</span></span></span></span> 段的第 <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.65952em;vertical-align:0"></span><span class="mord mathnormal">i</span></span></span></span> 到 <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">i+1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.74285em;vertical-align:-.08333em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:.2222222222222222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:.2222222222222222em"></span></span><span class="base"><span class="strut" style="height:.64444em;vertical-align:0"></span><span class="mord">1</span></span></span></span> 次变换将变为：</p><p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable width="100%"><mtr><mtd width="50%"></mtd><mtd><mrow><msubsup><mi mathvariant="bold">H</mi><mi>j</mi><mrow><mi>i</mi><mo>+</mo><mn>1</mn></mrow></msubsup><mo>=</mo><msup><mrow><mi mathvariant="normal">T</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">f</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">r</mi></mrow><mi>i</mi></msup><mo stretchy="false">(</mo><mrow><mo fence="true">[</mo><msubsup><mi mathvariant="bold">H</mi><mi>j</mi><mi>i</mi></msubsup><mo separator="true">,</mo><msup><mrow><mi mathvariant="bold">B</mi><mi mathvariant="bold">u</mi><mi mathvariant="bold">s</mi></mrow><mi>i</mi></msup><mo fence="true">]</mo></mrow><mo stretchy="false">)</mo></mrow></mtd><mtd width="50%"></mtd><mtd><mtext>(4)</mtext></mtd></mtr></mtable><annotation encoding="application/x-tex">\mathbf{H}_{j}^{i+1}=\mathrm{Transformer}^{i}(\left[\mathbf{H}_{j}^{i},\mathbf{Bus}^{i}\right])\tag4</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2777669999999999em;vertical-align:-.403103em"></span><span class="mord"><span class="mord"><span class="mord mathbf">H</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.874664em"><span style="top:-2.4330050000000005em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span></span></span></span><span style="top:-3.113em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.403103em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:.2777777777777778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:.2777777777777778em"></span></span><span class="base"><span class="strut" style="height:1.292112em;vertical-align:-.383108em"></span><span class="mord"><span class="mord"><span class="mord mathrm">T</span><span class="mord mathrm">r</span><span class="mord mathrm">a</span><span class="mord mathrm">n</span><span class="mord mathrm">s</span><span class="mord mathrm" style="margin-right:.07778em">f</span><span class="mord mathrm">o</span><span class="mord mathrm">r</span><span class="mord mathrm">m</span><span class="mord mathrm">e</span><span class="mord mathrm">r</span></span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.9090039999999999em"><span style="top:-3.1473400000000002em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span><span class="mopen">(</span><span class="minner"><span class="mopen delimcenter" style="top:0"><span class="delimsizing size1">[</span></span><span class="mord"><span class="mord"><span class="mord mathbf">H</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.874664em"><span style="top:-2.4530000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.05724em">j</span></span></span></span><span style="top:-3.1130000000000004em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.383108em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mord"><span class="mord"><span class="mord mathbf">B</span><span class="mord mathbf">u</span><span class="mord mathbf">s</span></span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.900674em"><span style="top:-3.1390100000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span><span class="mclose delimcenter" style="top:0"><span class="delimsizing size1">]</span></span></span><span class="mclose">)</span></span><span class="tag"><span class="strut" style="height:1.312107em;vertical-align:-.403103em"></span><span class="mord text"><span class="mord">(</span><span class="mord"><span class="mord">4</span></span><span class="mord">)</span></span></span></span></span></span></p><p>其中 “[]” 表示串联，而 “Transformeri (・)” 是 transformers 的第 i 层。最终的新闻嵌入是通过聚合最后一层的所有隐藏状态，即 H−1∗来获得的。在附录 A 中，我们给出了将 BusLM 应用于 transformers 中不同类型的层的实现细节。实验证明，该方法在时间效率和存储效率上都有明显的提高，同时，由于文本拆分而造成的信息损失也得到了充分的缓解。</p><h2 id="自回归用户建模"><a class="anchor" href="#自回归用户建模">#</a> 自回归用户建模</h2><p>本文提出了自回归用户建模，以便更有效地利用用户历史 (图 6)，其中可以在一次新闻编码中预测关于用户的所有新闻点击。不再逐例处理每个训练实例<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⟨</mo><msub><mo><mi mathvariant="normal">clicks</mi><mo>⁡</mo></mo><mrow><mo>=</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><msub><mo><mi mathvariant="normal">click</mi><mo>⁡</mo></mo><mrow><mo>≤</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy="false">⟩</mo></mrow><annotation encoding="application/x-tex">\langle\operatorname{clicks}_{=t},\operatorname{click}_{\leq t-1}\rangle</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mopen">⟨</span><span class="mop"><span class="mop"><span class="mord mathrm">c</span><span class="mord mathrm">l</span><span class="mord mathrm">i</span><span class="mord mathrm">c</span><span class="mord mathrm">k</span><span class="mord mathrm">s</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">=</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mop"><span class="mop"><span class="mord mathrm">c</span><span class="mord mathrm">l</span><span class="mord mathrm">i</span><span class="mord mathrm">c</span><span class="mord mathrm">k</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.301108em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">≤</span><span class="mord mathnormal mtight">t</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24517899999999998em"><span></span></span></span></span></span></span><span class="mclose">⟩</span></span></span></span>，而是将整个用户历史<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo><mi mathvariant="normal">clicks</mi><mo>⁡</mo></mo><mrow><mo>≤</mo><mi>L</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\operatorname{clicks}_{\leq L}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.939619em;vertical-align:-.24517899999999998em"></span><span class="mop"><span class="mop"><span class="mord mathrm">c</span><span class="mord mathrm">l</span><span class="mord mathrm">i</span><span class="mord mathrm">c</span><span class="mord mathrm">k</span><span class="mord mathrm">s</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.328331em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">≤</span><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24517899999999998em"><span></span></span></span></span></span></span></span></span></span> 作为一个统一的训练实例 (设<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.68333em;vertical-align:0"></span><span class="mord mathnormal">L</span></span></span></span> 为用户历史的最大长度)。训练器将对所有的历史新闻点击进行编码，得到新闻嵌入集:<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">{</mo><msub><mi mathvariant="bold-italic">θ</mi><mi>l</mi></msub><msub><mo stretchy="false">}</mo><mrow><mo>≤</mo><mi>L</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\{\boldsymbol{\theta}_l\}_{\le L}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mopen">{</span><span class="mord"><span class="mord"><span class="mord"><span class="mord boldsymbol" style="margin-right:.03194em">θ</span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.33610799999999996em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:.01968em">l</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.328331em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">≤</span><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24517899999999998em"><span></span></span></span></span></span></span></span></span></span>。然后训练器将计算用户嵌入集<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">{</mo><msub><mi mathvariant="bold-italic">μ</mi><mi>t</mi></msub><msub><mo stretchy="false">}</mo><mrow><mo>≤</mo><mi>L</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\{\boldsymbol{\mu}_t\}_{\le L}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mopen">{</span><span class="mord"><span class="mord"><span class="mord"><span class="mord boldsymbol">μ</span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.18641599999999994em"><span style="top:-2.4558600000000004em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24414em"><span></span></span></span></span></span></span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.328331em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">≤</span><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24517899999999998em"><span></span></span></span></span></span></span></span></span></span>，其中 μt 以前面的新闻嵌入<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">{</mo><msub><mi mathvariant="bold-italic">θ</mi><mi>l</mi></msub><msub><mo stretchy="false">}</mo><mrow><mo>≤</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\{\boldsymbol{\theta}_l\}_{\le t}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mopen">{</span><span class="mord"><span class="mord"><span class="mord"><span class="mord boldsymbol" style="margin-right:.03194em">θ</span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.33610799999999996em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:.01968em">l</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.295179em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">≤</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.24517899999999998em"><span></span></span></span></span></span></span></span></span></span> 为条件。每个用户嵌入用于预测下一个时间戳的新闻点击，其中一个样本用户的总体预测损失<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="script">L</mi><mrow><mi>a</mi><mi>u</mi><mi>t</mi><mi>o</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\mathcal{L}_{auto}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.83333em;vertical-align:-.15em"></span><span class="mord"><span class="mord"><span class="mord mathcal">L</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight">o</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span></span></span></span> w.r.t. 计算为:</p><p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable width="100%"><mtr><mtd width="50%"></mtd><mtd><mrow><msub><mi mathvariant="script">L</mi><mrow><mi>a</mi><mi>u</mi><mi>t</mi><mi>o</mi></mrow></msub><mo>=</mo><mo>−</mo><munder><mo>∑</mo><mrow><mi>t</mi><mo>&lt;</mo><mi>L</mi></mrow></munder><mi mathvariant="normal">log</mi><mo>⁡</mo><mfrac><mrow><mi mathvariant="normal">exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo stretchy="false">⟨</mo><msub><mi>θ</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo separator="true">,</mo><msub><mi>μ</mi><mi>t</mi></msub><mo stretchy="false">⟩</mo><mo stretchy="false">)</mo></mrow><mrow><mi mathvariant="normal">exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo stretchy="false">⟨</mo><msub><mi>θ</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo separator="true">,</mo><msub><mi>μ</mi><mi>t</mi></msub><mo stretchy="false">⟩</mo><mo stretchy="false">)</mo><mo>+</mo><munder><mo>∑</mo><msubsup><mi>θ</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup></munder><mi mathvariant="normal">exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo stretchy="false">⟨</mo><msubsup><mi>θ</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow><msup><mrow></mrow><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msubsup><mo separator="true">,</mo><msub><mi>μ</mi><mi>t</mi></msub><mo stretchy="false">⟩</mo><mo stretchy="false">)</mo></mrow></mfrac></mrow></mtd><mtd width="50%"></mtd><mtd><mtext>(5)</mtext></mtd></mtr></mtable><annotation encoding="application/x-tex">\mathcal{L}_{a u t o}=-\sum_{t&lt;L}\operatorname{log}\frac{\operatorname{exp}(\langle\theta_{t+1},\mu_{t}\rangle)}{\operatorname{exp}(\langle\theta_{t+1},\mu_{t}\rangle)+\sum_{\theta_{t+1}^{\prime}}\operatorname{exp}(\langle\theta_{t+1}^{&#x27;},\mu_{t}\rangle)}\tag5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:.83333em;vertical-align:-.15em"></span><span class="mord"><span class="mord"><span class="mord mathcal">L</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight">o</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:.2777777777777778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:.2777777777777778em"></span></span><span class="base"><span class="strut" style="height:2.7487060000000003em;vertical-align:-1.321706em"></span><span class="mord">−</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.050005em"><span style="top:-1.8556639999999998em;margin-left:0"><span class="pstrut" style="height:3.05em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mrel mtight">&lt;</span><span class="mord mathnormal mtight">L</span></span></span></span><span style="top:-3.0500049999999996em"><span class="pstrut" style="height:3.05em"></span><span><span class="mop op-symbol large-op">∑</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.321706em"><span></span></span></span></span></span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mop"><span class="mord mathrm">l</span><span class="mord mathrm">o</span><span class="mord mathrm" style="margin-right:.01389em">g</span></span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.427em"><span style="top:-2.28722em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mop"><span class="mord mathrm">e</span><span class="mord mathrm">x</span><span class="mord mathrm">p</span></span><span class="mopen">(</span><span class="mopen">⟨</span><span class="mord"><span class="mord mathnormal" style="margin-right:.02778em">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.301108em"><span style="top:-2.5500000000000003em;margin-left:-.02778em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.208331em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mord"><span class="mord mathnormal">μ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mclose">⟩</span><span class="mclose">)</span><span class="mspace" style="margin-right:.2222222222222222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:.2222222222222222em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:-.0000050000000000050004em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.21943000000000001em"><span style="top:-2.40029em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:.02778em">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.7416285714285715em"><span style="top:-2.1884857142857146em;margin-left:-.02778em;margin-right:.07142857142857144em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span><span style="top:-2.8448em;margin-right:.07142857142857144em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.3710357142857143em"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.559435em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mop"><span class="mord mathrm">e</span><span class="mord mathrm">x</span><span class="mord mathrm">p</span></span><span class="mopen">(</span><span class="mopen">⟨</span><span class="mord"><span class="mord mathnormal" style="margin-right:.02778em">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.82278em"><span style="top:-2.433692em;margin-left:-.02778em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.0448000000000004em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span></span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.6828285714285715em"><span style="top:-2.786em;margin-right:.07142857142857144em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.3246389999999999em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mord"><span class="mord mathnormal">μ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mclose">⟩</span><span class="mclose">)</span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:.04em"></span></span><span style="top:-3.677em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mop"><span class="mord mathrm">e</span><span class="mord mathrm">x</span><span class="mord mathrm">p</span></span><span class="mopen">(</span><span class="mopen">⟨</span><span class="mord"><span class="mord mathnormal" style="margin-right:.02778em">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.301108em"><span style="top:-2.5500000000000003em;margin-left:-.02778em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.208331em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mord"><span class="mord mathnormal">μ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.2805559999999999em"><span style="top:-2.5500000000000003em;margin-left:0;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.15em"><span></span></span></span></span></span></span><span class="mclose">⟩</span><span class="mclose">)</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.272215em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span><span class="tag"><span class="strut" style="height:2.7487060000000003em;vertical-align:-1.321706em"></span><span class="mord text"><span class="mord">(</span><span class="mord"><span class="mord">5</span></span><span class="mord">)</span></span></span></span></span></span></p><p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⟨</mo><mo separator="true">⋅</mo><mo stretchy="false">⟩</mo></mrow><annotation encoding="application/x-tex">\langle·\rangle</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-.25em"></span><span class="mopen">⟨</span><span class="mpunct">⋅</span><span class="mspace" style="margin-right:.16666666666666666em"></span><span class="mclose">⟩</span></span></span></span> 计算用户和新闻嵌入的相关性，例如内积；<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>θ</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow><msup><mrow></mrow><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msubsup></mrow><annotation encoding="application/x-tex">\theta_{t+1}^{&#x27;}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.248919em;vertical-align:-.30643899999999996em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:.02778em">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:.94248em"><span style="top:-2.451892em;margin-left:-.02778em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.063em;margin-right:.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span></span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:.8278285714285715em"><span style="top:-2.931em;margin-right:.07142857142857144em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:.30643899999999996em"><span></span></span></span></span></span></span></span></span></span> 是负样本的嵌入。</p><p>paper：<span class="exturl" data-url="aHR0cHM6Ly9hcnhpdi5vcmcvYWJzLzIxMDIuMDkyNjg=">https://arxiv.org/abs/2102.09268</span></p><p>codes：<span class="exturl" data-url="aHR0cHM6Ly9naXRodWIuY29tL01pY3Jvc29mdC9TcGVlZHlSZWM=">https://github.com/Microsoft/SpeedyRec</span></p><div class="tags"><a href="/tags/%E6%8E%A8%E8%8D%90%E7%B3%BB%E7%BB%9F/" rel="tag"><i class="ic i-tag"></i> 推荐系统</a> <a href="/tags/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" rel="tag"><i class="ic i-tag"></i> 新闻推荐</a> <a href="/tags/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" rel="tag"><i class="ic i-tag"></i> 论文精读</a> <a href="/tags/%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B/" rel="tag"><i class="ic i-tag"></i> 预训练模型</a></div></div><footer><div class="meta"><span class="item"><span class="icon"><i class="ic i-calendar-check"></i> </span><span class="text">更新于</span> <time title="修改时间：2023-04-21 14:00:36" itemprop="dateModified" datetime="2023-04-21T14:00:36+08:00">2023-04-21</time> </span><span id="posts/c51ea00e/" class="item leancloud_visitors" data-flag-title="SpeedyFeed：用预训练语言模型训练大规模新闻推荐器" title="阅读次数"><span class="icon"><i class="ic i-eye"></i> </span><span class="text">阅读次数</span> <span class="leancloud-visitors-count"></span> <span class="text">次</span></span></div><div class="reward"><button><i class="ic i-heartbeat"></i> 赞赏</button><p>请我喝[茶]~(￣▽￣)~*</p><div id="qr"><div><img data-src="/images/wechatpay.png" alt="hang shun 微信支付"><p>微信支付</p></div><div><img data-src="/images/alipay.png" alt="hang shun 支付宝"><p>支付宝</p></div><div><img data-src="/images/paypal.png" alt="hang shun 贝宝"><p>贝宝</p></div></div></div><div id="copyright"><ul><li class="author"><strong>本文作者： </strong>hang shun <i class="ic i-at"><em>@</em></i>航 順</li><li class="link"><strong>本文链接：</strong> <a href="https://jiang-hs.gitee.io/posts/c51ea00e/" title="SpeedyFeed：用预训练语言模型训练大规模新闻推荐器">https://jiang-hs.gitee.io/posts/c51ea00e/</a></li><li class="license"><strong>版权声明： </strong>本站所有文章除特别声明外，均采用 <span class="exturl" data-url="aHR0cHM6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LW5jLXNhLzQuMC9kZWVkLnpo"><i class="ic i-creative-commons"><em>(CC)</em></i>BY-NC-SA</span> 许可协议。转载请注明出处！</li></ul></div></footer></article></div><div class="post-nav"><div class="item left"><a href="/posts/3cbd0624/" itemprop="url" rel="prev" data-background-image="https:&#x2F;&#x2F;pic1.imgdb.cn&#x2F;item&#x2F;64427c3a0d2dde5777afad9b.jpg" title="GRU4Rec：基于循环神经网络的会话推荐"><span class="type">上一篇</span> <span class="category"><i class="ic i-flag"></i> 序列推荐</span><h3>GRU4Rec：基于循环神经网络的会话推荐</h3></a></div><div class="item right"><a href="/posts/e3b95b70/" itemprop="url" rel="next" data-background-image="https:&#x2F;&#x2F;pic1.imgdb.cn&#x2F;item&#x2F;64427c370d2dde5777afaa63.jpg" title="Fastformer：Additive Attention Can Be All You Need"><span class="type">下一篇</span> <span class="category"><i class="ic i-flag"></i> 隐私保护新闻推荐</span><h3>Fastformer：Additive Attention Can Be All You Need</h3></a></div></div><div class="wrap" id="comments"></div></div><div id="sidebar"><div class="inner"><div class="panels"><div class="inner"><div class="contents panel pjax" data-title="文章目录"><ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#%E4%BB%8B%E7%BB%8D"><span class="toc-number">1.</span> <span class="toc-text">介绍</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#%E7%9B%B8%E5%85%B3%E5%B7%A5%E4%BD%9C"><span class="toc-number">2.</span> <span class="toc-text">相关工作</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B"><span class="toc-number">2.1.</span> <span class="toc-text">预训练模型</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#%E6%96%B9%E6%B3%95"><span class="toc-number">3.</span> <span class="toc-text">方法</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#%E9%9B%86%E4%B8%AD%E5%BC%8F%E6%96%B0%E9%97%BB%E7%BC%96%E7%A0%81"><span class="toc-number">3.1.</span> <span class="toc-text">集中式新闻编码</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E7%BC%93%E5%AD%98%E5%8A%A0%E9%80%9F%E6%96%B0%E9%97%BB%E7%BC%96%E7%A0%81"><span class="toc-number">3.2.</span> <span class="toc-text">缓存加速新闻编码</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E6%80%BB%E7%BA%BF%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B"><span class="toc-number">3.3.</span> <span class="toc-text">总线语言模型</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E8%87%AA%E5%9B%9E%E5%BD%92%E7%94%A8%E6%88%B7%E5%BB%BA%E6%A8%A1"><span class="toc-number">3.4.</span> <span class="toc-text">自回归用户建模</span></a></li></ol></li></ol></div><div class="related panel pjax" data-title="系列文章"><ul><li class="active"><a href="/posts/c51ea00e/" rel="bookmark" title="SpeedyFeed：用预训练语言模型训练大规模新闻推荐器">SpeedyFeed：用预训练语言模型训练大规模新闻推荐器</a></li><li><a href="/posts/a05e9ec4/" rel="bookmark" title="Tiny-NewsRec：高效的基于预训练模型的新闻推荐">Tiny-NewsRec：高效的基于预训练模型的新闻推荐</a></li></ul></div><div class="overview panel" data-title="站点概览"><div class="author" itemprop="author" itemscope itemtype="http://schema.org/Person"><img class="image" itemprop="image" alt="hang shun" data-src="/images/avatar.jpg"><p class="name" itemprop="name">hang shun</p><div class="description" itemprop="description">世中逢尔，雨中逢花</div></div><nav class="state"><div class="item posts"><a href="/archives/"><span class="count">45</span> <span class="name">文章</span></a></div><div class="item categories"><a href="/categories/"><span class="count">10</span> <span class="name">分类</span></a></div><div class="item tags"><a href="/tags/"><span class="count">25</span> <span class="name">标签</span></a></div></nav><div class="social"><span class="exturl item github" data-url="aHR0cHM6Ly9naXRodWIuY29tL0pJQU5HLUhT" title="https:&#x2F;&#x2F;github.com&#x2F;JIANG-HS"><i class="ic i-github"></i></span> <span class="exturl item zhihu" data-url="aHR0cHM6Ly93d3cuemhpaHUuY29tL3Blb3BsZS9odWktc2h1bi14aW4tbGl1" title="https:&#x2F;&#x2F;www.zhihu.com&#x2F;people&#x2F;hui-shun-xin-liu"><i class="ic i-zhihu"></i></span> <span class="exturl item music" data-url="aHR0cHM6Ly9tdXNpYy4xNjMuY29tLyMvdXNlci9ob21lP2lkPTE4MzkwMTczMzI=" title="https:&#x2F;&#x2F;music.163.com&#x2F;#&#x2F;user&#x2F;home?id&#x3D;1839017332"><i class="ic i-cloud-music"></i></span> <span class="exturl item bilibili" data-url="aHR0cHM6Ly9zcGFjZS5iaWxpYmlsaS5jb20vMzIxMTYyNDg1" title="https:&#x2F;&#x2F;space.bilibili.com&#x2F;321162485"><i class="ic i-bilibili"></i></span></div><ul class="menu"><li class="item"><a href="/" rel="section"><i class="ic i-home"></i>首页</a></li><li class="item"><a href="/about/" rel="section"><i class="ic i-user"></i>关于</a></li><li class="item dropdown"><a href="javascript:void(0);"><i class="ic i-feather"></i>文章</a><ul class="submenu"><li class="item"><a href="/archives/" rel="section"><i class="ic i-list-alt"></i>归档</a></li><li class="item"><a href="/categories/" rel="section"><i class="ic i-th"></i>分类</a></li><li class="item"><a href="/tags/" rel="section"><i class="ic i-tags"></i>标签</a></li></ul></li><li class="item"><a href="/friends/" rel="section"><i class="ic i-heart"></i>友達</a></li><li class="item"><a href="/movie/" rel="section"><i class="ic i-play"></i>movie</a></li><li class="item"><a href="/music/" rel="section"><i class="ic i-music"></i>music</a></li></ul></div></div></div><ul id="quick"><li class="prev pjax"><a href="/posts/3cbd0624/" rel="prev" title="上一篇"><i class="ic i-chevron-left"></i></a></li><li class="up"><i class="ic i-arrow-up"></i></li><li class="down"><i class="ic i-arrow-down"></i></li><li class="next pjax"><a href="/posts/e3b95b70/" rel="next" title="下一篇"><i class="ic i-chevron-right"></i></a></li><li class="percent"></li></ul></div></div><div class="dimmer"></div></div></main><footer id="footer"><div class="inner"><div class="widgets"><div class="rpost pjax"><h2>随机文章</h2><ul><li class="item"><div class="breadcrumb"></div><span><a href="/posts/ae4d52e6/" title="顺序价格机制的强化学习">顺序价格机制的强化学习</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" title="分类于 论文精读">论文精读</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E5%BA%8F%E5%88%97%E6%8E%A8%E8%8D%90/" title="分类于 序列推荐">序列推荐</a></div><span><a href="/posts/3cbd0624/" title="GRU4Rec：基于循环神经网络的会话推荐">GRU4Rec：基于循环神经网络的会话推荐</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" title="分类于 论文精读">论文精读</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 新闻推荐">新闻推荐</a> <i class="ic i-angle-right"></i> <a href="/categories/%E6%B3%A8%E6%84%8F%E5%8A%9B%E6%9C%BA%E5%88%B6/" title="分类于 注意力机制">注意力机制</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/%E9%9A%90%E7%A7%81%E4%BF%9D%E6%8A%A4%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 隐私保护新闻推荐">隐私保护新闻推荐</a></div><span><a href="/posts/e3b95b70/" title="Fastformer：Additive Attention Can Be All You Need">Fastformer：Additive Attention Can Be All You Need</a></span></li><li class="item"><div class="breadcrumb"></div><span><a href="/posts/b3064b8/" title="梯度下降及线性回归详解">梯度下降及线性回归详解</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/Debug/" title="分类于 Debug">Debug</a></div><span><a href="/posts/435711c9/" title="解决github无法访问">解决github无法访问</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" title="分类于 论文精读">论文精读</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 新闻推荐">新闻推荐</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/%E9%9A%90%E7%A7%81%E4%BF%9D%E6%8A%A4%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 隐私保护新闻推荐">隐私保护新闻推荐</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%81%94%E9%82%A6%E5%AD%A6%E4%B9%A0/" title="分类于 联邦学习">联邦学习</a></div><span><a href="/posts/d414c6e0/" title="Efficient-FedRec：高效的新闻推荐隐私保护框架">Efficient-FedRec：高效的新闻推荐隐私保护框架</a></span></li><li class="item"><div class="breadcrumb"></div><span><a href="/posts/9dc5ce37/" title="DIB-PEB-Sequientia-RS笔记">DIB-PEB-Sequientia-RS笔记</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" title="分类于 论文精读">论文精读</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 新闻推荐">新闻推荐</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/%E7%94%A8%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B%E5%81%9A%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 用预训练模型做新闻推荐">用预训练模型做新闻推荐</a></div><span><a href="/posts/c51ea00e/" title="SpeedyFeed：用预训练语言模型训练大规模新闻推荐器">SpeedyFeed：用预训练语言模型训练大规模新闻推荐器</a></span></li><li class="item"><div class="breadcrumb"></div><span><a href="/posts/c1da6b2c/" title="贝叶斯个性化排序-BPR">贝叶斯个性化排序-BPR</a></span></li><li class="item"><div class="breadcrumb"><a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/" title="分类于 论文精读">论文精读</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 新闻推荐">新闻推荐</a> <i class="ic i-angle-right"></i> <a href="/categories/%E8%AE%BA%E6%96%87%E7%B2%BE%E8%AF%BB/%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/%E7%94%A8%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B%E5%81%9A%E6%96%B0%E9%97%BB%E6%8E%A8%E8%8D%90/" title="分类于 用预训练模型做新闻推荐">用预训练模型做新闻推荐</a></div><span><a href="/posts/a05e9ec4/" title="Tiny-NewsRec：高效的基于预训练模型的新闻推荐">Tiny-NewsRec：高效的基于预训练模型的新闻推荐</a></span></li></ul></div><div><h2>最新评论</h2><ul class="leancloud-recent-comment"></ul></div></div><div class="status"><div class="copyright">&copy; 2020 – <span itemprop="copyrightYear">2023</span> <span class="with-love"><i class="ic i-sakura rotate"></i> </span><span class="author" itemprop="copyrightHolder">hang shun @ hang shun</span></div><div class="count"><span class="post-meta-item-icon"><i class="ic i-chart-area"></i> </span><span title="站点总字数">267k 字</span> <span class="post-meta-divider">|</span> <span class="post-meta-item-icon"><i class="ic i-coffee"></i> </span><span title="站点阅读时长">4:02</span></div><div class="powered-by">基于 <span class="exturl" data-url="aHR0cHM6Ly9oZXhvLmlv">Hexo</span> & Theme.<span class="exturl" data-url="aHR0cHM6Ly9naXRodWIuY29tL2FtZWhpbWUvaGV4by10aGVtZS1zaG9rYQ==">Shoka</span></div></div></div></footer></div><script data-config type="text/javascript">var LOCAL={path:"posts/c51ea00e/",favicon:{show:"(´Д｀)被发现了！",hide:"（●´3｀●）我藏好了~"},search:{placeholder:"文章搜索",empty:"关于 「 ${query} 」，什么也没搜到",stats:"${time} ms 内找到 ${hits} 条结果"},valine:!0,copy_tex:!0,katex:!0,fancybox:!0,copyright:'复制成功，转载请遵守 <i class="ic i-creative-commons"></i>BY-NC-SA 协议。',ignores:[function(e){return e.includes("#")},function(e){return new RegExp(LOCAL.path+"$").test(e)}]}</script><script src="https://cdn.polyfill.io/v2/polyfill.js"></script><script src="//cdn.jsdelivr.net/combine/npm/pace-js@1.0.2/pace.min.js,npm/pjax@0.2.8/pjax.min.js,npm/whatwg-fetch@3.4.0/dist/fetch.umd.min.js,npm/animejs@3.2.0/lib/anime.min.js,npm/algoliasearch@4/dist/algoliasearch-lite.umd.js,npm/instantsearch.js@4/dist/instantsearch.production.min.js,npm/lozad@1/dist/lozad.min.js,npm/quicklink@2/dist/quicklink.umd.js"></script><script src="/js/app.js?v=0.0.0"></script></body></html><!-- rebuild by hrmmi -->